DOCUMENT RESUME 



ED 470 918 



JC 030 002 



AUTHOR 
TITLE ■ 

PUB DATE 
NOTE 

AVAILABLE FROM 
PUB TYPE 
EDRS PRICE 
DESCRIPTORS 



IDENTIFIERS 



Kennedy, Cathleen A. 

Evaluation of Online Curriculum at College of San Mateo. 

1998-12-00 

7p. 

For full text: http://smccd.net/accounts/kennedyc/ rsch/. 
Reports - Research (143) — Tests /Questionnaires (160) 
EDRS Price MF01/PC01 Plus Postage. 

^Community Colleges; Comparative Analysis; Distance 
Education; ^Online Courses;. ^Program Evaluation; ^Student 
Attitudes; Student Surveys; Two Year Colleges 
^College of San Mateo CA 



ABSTRACT 

The goal of this project was to evaluate the College of San 
Mateo's (California) current online program to ascertain how well it met the 
needs of current and prospective students. This initial evaluation was 
formative, focusing on how the institution could improve what it started. The 
author evaluated student satisfaction with online course methods and content; 
she also' gathered attitude data from students who were not currently taking 
online courses, in an attempt to determine if an audience existed for more 
online offerings. Two surveys-one with Likert-type questions and one with 
open-ended questions-were administered to students; a total of 139 surveys 
were administered and collected during class time, and 20 surveys were 
completed by students online . One finding was that a substantial percentage 
of current students would consider taking online courses. This was 
particularly true of evening students. A second finding was that students 
valued good teaching and teachers more than other aspects of courses. One 
unexpected finding was that both on-campus and online students felt that the 
teacher was the most important factor for success in an on-campus course, 
while student study habits was the most important factor in online courses. 
Finally, online students rated their enjoyment of courses at nearly the same 
level as on-campus students. The Online Needs Assessment Survey is appended. 
(Author/EMH) ■- 
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§ Abstract 



The goal of this project was to evaluate the current online program to get 
a sense of how well it meets the needs of current and prospective 
students. This initial evaluation was formative, focusing on how the 
institution can improve what it's started. I evaluated student satisfaction 
with online course methods and content, and gathered attitude data from 
students who are not currently taking online courses in an initial attempt 
to determine if an audience exists for more online courses. 
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One finding is that a substantial percentage of current students would 
consider taking online courses. This was particularly true of evening 
students. A second finding is that students value good teaching and 
teachers more than other aspects of courses. A surprising finding was 
that both on-campus and online students felt that "the teacher" was the 
most important factor for success in an on-campus course, while 
"student study habits" was the most important factor in online courses. 
Finally, online students rated their enjoyment of courses at nearly the 
same level as on-campus students. 



An abbreviated version of the final report follows. 
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This evaluation has two primary components: evaluating the need for the 
program and evaluating current program practices. Since this program is 
still relatively new, the results of the evaluation will inform decisions 
about the future of the program. The overall design, then, will be 
formative. This section of the report is organized around the three 
evaluation questions. 

Ql. Are online courses needed? 



A survey of eight open-ended questions was administered to students in 
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day and evening on-campus classes at CSM in November 1998 to 
ascertain their perceived need for online courses and their interest in 
taking courses online. Students also reported demographic information, 
information about their academic background and experience in non- 
traditional course formats, their experience using computers, and their 
attitudes about using computers (see the Online Needs Assessment 
Survey in Appendix A). Four instructors were given an opportunity to 
conduct the surveys in their classes; two participated. 54 surveys were 
administered and collected by the instructors during class time. 

A potential bias is introduced by collecting data from current students. 
Since these individuals are already served by traditional on-campus 
courses, unlike potential students who cannot or do not attend traditional 
courses, they may be biased against the need for online courses. This 
may be counterbalanced, or even totally offset, by the fact that current 
students are aware of the availability of the wide variety of courses at 
CSM, some of which may not be offered at times they can attend. 

An interesting limitation to this part of the study is gathering data about 
online courses from those who do not know what they are. 

Q2. What do students value in courses? 

Data for answering this question comes from two sources: the survey for 
Question 1 above, and another survey administered to students in both 
traditional and online courses (see the Course Evaluation Survey in 
Appendix A). This second survey is comprised of three open-ended 
questions and seven Likert-type questions with accompanying open- 
ended items for explanations. The survey explores student attitudes and 
perceptions about what they like and dislike about courses. Questions 
were framed to elicit responses indicative of student engagement in the 
learning process. Eight instructors who teach both on-campus and online 
courses were given an opportunity to conduct the surveys in their 
or ask their online students to complete a web-based version of the 
survey; five teachers participated, making the survey available to a total 
of twelve on-campus and online courses. 85 surveys were administered 
and collected by the instructors during class time and 20 surveys were 
completed by students online. 

The skewed number of responses from on-campus students over online 
students introduces significant bias into the results and reduces the 
significance of the findings for anything other than general guidelines for 
further study. In addition, the more voluntary nature of completing the 
online survey will quite likely bias the results towards opinions of those 
who are more actively engaged, and probably more satisfied, in their 
online course. For online students, the results are probably heavily 
biased towards reporting what satisfied students like about their courses, 
which is helpful, but not as complete as also learning what dissatisfied 
students think about their online courses. 

Q3. Do current online courses include the features students value? 

Data for answering this question also comes from the Course Evaluation 
Survey (see Appendix A). The relevant data are the 20 responses from 
online students on the Likert-type items described above. Again, I found 
a strong bias towards positive responses about online courses, but the 
open-ended items were helpful in analyzing what students value most in 
online courses. 
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Instrument Design 

The surveys were intentionally designed with significant opportunity for 
respondents to describe their attitudes and explain their answers to 
reduce the amount of bias closed-end questions introduce into items. I 
was particularly interested in gaining insights about what students like 
and dislike about courses that we hadn't thought of in preliminary 
planning. 

The attitude measures used in the Course Evaluation Survey are ordinal, 
suggesting a pattern of order, but not of true magnitude. I used 
distribution to determine general attitude patterns of students in on 
campus and online courses. The variables I studied included: enjoyment 
of a course, engagement in the subject matter, satisfaction with 
interactions with the teacher and with other students, a sense that the 
course components are well-integrated, whether the course workload is 
appropriate, and whether students would recommend the course to their 
friends. There are four measurement classes for each variable: strongly 
agree, agree, disagree, and strongly disagree. 

The Course Evaluation Survey was analyzed with Item Response Theory 
(IRT) to help determine validity and reliability. The analysis revealed an 
internal consistency of 0.72, which is sufficient for instruments with a 
small number of items. The Online Needs Assessment Survey had a 
much smaller value for internal consistency, 0.26, although all items had 
reasonable INFIT t values. This may be a result of scoring the surveys, 
since it is comprised of open ended items. 

Analysis 

Although a true experimental design would be valuable, particularly for a 
summative evaluation, I was unable to use any method that even 
approximates randomization. The primary reason for this is that the 
College does not have authority to deny students access to online 
courses. As a result, student selection does not come close to random 
because students who self-select into online courses probably have 
different characteristics than students who don't. 

Given this constraint, the analysis is largely correlative, searching for 
trends that indicate strong student preferences. Multiple regression was 
run on several determinant variables with various sets of independent 
variables to determine which had the greatest influence. In other cases, 
simple frequencies were adequate for determining trends for the purpose 
of formative evaluation of current practices. The following analyses were 
performed on the data: 



Evaluation Results 



Interpretations 

These findings suggest that there is a need for providing more online 
courses, despite biases in the sample. Although this sample includes 
only current college students, the findings provide insight into the most 
prevalent reasons for taking courses. This information can help CSM 
identify unserved population groups that could benefit from additional 
educational opportunities provided in an online format. The finding that 
52% of the students surveyed would consider taking an online course 
suggests that we should do more research of the population at large in 
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this regard, since current students may not be indicative of non-students' 
attitudes. Students' reasons for not considering taking online courses are 
nevertheless informative. Some believe they would not have as much 
teacher contact as they have in traditional courses; others just do not 
know how an online course is conducted. These findings suggest that 
others might consider online courses if they knew more about them. 

In determining what students value in courses, I considered what 
students reported as "best" aspects of their courses, and what students 
felt were the most important factors for student success. Most students 
understood the importance of good study habits for success in both on- 
campus and online courses, and rated this factor as most important for 
online courses. It is interesting, however, that very few students thought 
teacher quality was an important factor in online courses, while they 
it the highest importance as a factor for success in on-campus courses. 
Even students familiar with online courses do not report that the teacher 
of an online course is a critical factor in student success. Current online 
students echo this finding by rating course schedule and flexibility as 
more important than the teacher in online courses. This may be because 
students are unaware of the teacher's role in an online course, or 
because teachers do not, in fact, influence student success, or it may 
reflect a bias in the sample of 20 online students. This issue could be 
illuminated by further study. In addition, CSM could improve the potential 
for success in online courses by helping students develop good study 
habits. 

Although the vast majority of online students reported positive attitudes 
about their experience in online courses, this study did not provide strong 
evidence that current online courses meet the needs of students. First, 
the number of online students who participated was small, and we do not 
know whether these students are representative of the entire population 
of online students at CSM. In addition, although the Course Evaluation 
surveys completed by on-campus students include a wealth of 
handwritten remarks elaborating the reasons students were satisfied or 
dissatisfied with their courses, students who submitted the surveys 
were much more brief in their remarks. Again, the preliminary findings 
suggest that additional study be conducted to gain a better 
understanding of what factors contribute most to online students' 
satisfaction with their courses. 

Conclusion 



Criteria 

The online education program at College of San Mateo was evaluated 
with respect to meeting the needs of current and potential students. 
Much of the analysis was exploratory to determine what contributes to 
students' perceived need for online courses, what students value in their 
educational experiences, and what factors contribute to students 
becoming engaged in the process of learning. The study also evaluated 
attitude responses from a small sample of online students to get an idea 
of what course features online students appreciate. 

Although the study was significantly limited by the composition of the 
sample population, an important purpose was served in testing the two 
assessment instruments. Both instruments were found to have construct 
validity and their results can be considered reliable with moderately high 
values for separation reliability and Cronbach's alpha. The population 
bias most likely alters implications of the study, but has less of an impact 
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on testing the measurement tools. 

The findings of this evaluation can be used to enhance teacher training 
production and use of online courses and to guide efforts to inform and 
educate students and the public about online education. As a formative 
evaluation, this study meets its goals. 

Judgments 



Q1. Are online courses needed? 

This study indicates a substantial percentage of students would consider 
taking online courses if they were offered. Evening students in particular 
would consider taking courses online. The study did not solicit input from 
individuals who are not current students, which would provide more 
conclusive evidence of the need for more online courses if such a need 
does exist in that population. The study also did not evaluate enrollment 
patterns at CSM or at other local institutions to determine whether online 
programs have had adequate enrollments thus far. 

These findings provide evidence that further study should be conducted 
to determine whether individuals other than current students would 
benefit from college courses and would consider enrolling in online 
courses at CSM. 

Q2. What do students value in courses? 

Overall, students value good teaching and good teachers most of all. 
Even online students rated interactions with the instructor as very 
important for student engagement in the learning process. Students also 
rated the course schedule, course content (or subject matter) and hands- 
on course activities as features they like best in both on-campus and 
online courses. 

Students also agreed that student study habits are an important factor for 
success in on campus and online courses. Overall, students agreed that 
student study habits was the most important factor leading to success in 
online courses. 

03. Do current online courses include the features students value? 

Online students gave very positive ratings to their courses. A majority of 
online students would recommend the online course they are taking to a 
friend, and felt that they had sufficient interaction with their instructors. 
Online students rated their enjoyment of online courses at nearly the 
same level as on-campus students, and rated their engagement in online 
courses significantly higher than did on-campus students. 47% of online 
students versus 30% of on-campus students strongly agreed that the 
course engaged them in the subject matter. This indicates that CSM has 
used online courses to more successfully engage students in the 
learning process. Further research is suggested to validate these 
findings with a larger sample size, and to evaluate the impact of this on 
student outcomes. 

Recommendations 

Students who would not consider enrolling in online courses have some 
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distinct concerns about the learning experience and student outcomes in 
that environment. These concerns should be addressed. Further study 
should be conducted to determine if student outcomes are similar in day 
and evening on-campus courses and online courses. There is some 
concern that online courses might undermine enrollment in evening 
classes. This should also be explored further, in part, to determine if this 
is a "problem" or not. 

This study introduced significant bias through the process of selecting 
participants in the surveys. This bias can be minimized by conducting a 
larger study with more teachers and students participating, and by 
making both on-campus and online participation mandatory. A larger 
number of participating online students is essential to validate these 
preliminary findings. 

Additional content validity can be achieved by expanding the discussion 
of evaluation goals to more of the stakeholders. It would also be helpful 
to establish more criteria validity by relating student evaluation of their 
engagement in the course with actual performance. Predictive correlation 
should be conducted to ascertain whether student engagement is 
predictive of success in online courses. Further study could also explore 
what student engagement is, and whether it is more or less important 
than student study habits or teacher influence. These findings would be 
very important in guiding future development of online programs. 
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Online Needs Assessment Survey 



1. Gender 


2. Marital 


3. Children under 18? 


Male _ Female 


_ Single _Married 


_ Yes _ No 


4. Occupational Status 


5. Educational Status 


6. Types of Classes You Have Taken 


Full time 


Full time 


(check all that apply) 


Part time 


Part time 


_ Regular classroom courses 


_Not employed 


_ Not a student 


_ Correspondence courses 
_ Telecourses 
_ Online courses 


7. Education Level (completed) 


8. Age 


9. Ethnic Origin 


High school diploma 


17 or under 


_ American Indian or Alaskan Native 


_ AA/AS Degree 


18-22 


_ Asian or Pacific Islander 


_Certificate Program 


23-25 


_ Black, non-Hispanic 


_ BA/BS Degree 


26-35 


_ Hispanic 


_ M A/MS Degree 


_ 36-45 


_ White, non-Hispanic 


_Ed.D./Ph.D. Degree 
_ Other 

10. Experience Using Computers 

(check all that apply) 

_ Beginner 

Regular use of one or two applications 
_ Regular use of email 
_ Regular Web Access 
_ Experienced computer user 
Other 


_46 or over 


Other 




11. Attitude About Using Computers 

(check all that apply) 

_ I prefer not to use a computer 
_ I don’t mind using a computer 
_ I feel awkward using a computer 
I enjoy using a computer 
_ I would like to learn more 
Other 



13. Is there any reason for you to consider taking a college course now? Please explain. 

14. If a college course that you are interested in taking was offered during the day, would you bejable to take it? 
Why or why not? 

If it was offered at night would you be able to take it? Why or why not? 

If it was offered online would you be able to take it? Why or why not? 

15. If a college course that you are interested in taking was offered online, would you consider taking it? What 
other issues would enter into your decision? 

16. What do you think is the most important factor influencing your success in a course that you might take on 
campus? (i.e. the teacher, your study habits, the textbook, etc.) Why? 

17. What do you think is the most important factor influencing your success in a course that you might take 
online? Why? 

18. Are there any courses you would consider taking online, if you had the computer equipment at home and knew 
how to use it for taking courses? If yes, please describe the courses. If no, please explain why. 



Course Evaluation Survey 



1. Gender 

_ Male _ Female 

4. Occupational Status 

_ Full time 
_ Part time 
_Not employed 



7. Education Level (completed) 

_ High school diploma 
_ AA/AS Degree 
_Certificate Program 
_ BA/BS Degree 
_ MA/MS Degree 
_Ed.D./Ph.D. Degree 
Other 



2. Marital 3. Children under 18? 

_ Single _Married _Yes_No 



5. Educational Status 6. Types of Classes You Have Taken 



Full time 
Part time 
Not a student 



8. Age 

_ 17 or under 
_ 1 8 - 22 
_ 23-25 
_ 26-35 
_ 36-45 
46 or over 



(check all that apply) 

_ Regular classroom courses 
_ Correspondence courses 
_ Telecourses 
_ Online courses 

9. Ethnic Origin 

_ American Indian or Alaskan Native 
_ Asian or Pacific Islander 
_ Black, non-Hispanic 
_ Hispanic 

_ White, non-Hispanic 
Other 



10. Why did you enroll in this section of the course? (check all that apply) 

_ Schedule is good for me 
_ Teacher I wanted 
_ Friends are in the same class 
_ Other sections were full 
_ This was the only section offered 
Other reasons 



11. What did you like best about this course? 

12. What would you recommend changing in this course? 

13. If you have taken both on campus and online courses, how would you compare the two experiences? 



Strongly Agree Disagree Strongly 
Agree Disagree 



14. 1 enjoyed taking this course. 

15. This course engaged me in the subject matter and made me want to study 
it more. 

16. 1 had sufficient interaction and communication with my instructor. 

17. 1 had sufficient interaction and communication with other students. 

18. The course components were well-organized and effective. 

19. The course workload was appropriate. 

20. 1 would recommend this course to a friend. 
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